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ABSTRACT 

This article describes the work undertaken 
by a cluster of Resource Teachers: Learning 
and Behaviour (RTLB) to ensure that annual 
effectiveness reviews were more than a 
compliance exercise but a genuine attempt to 
improve their service to schools, hence the title of 
this article of "Going Beyond Audit". Historically, 
the cluster had met the reporting requirements 
set by the Ministry of Education (MOE), however 
to make the process more purposeful, reviews 
were broadened through the collection of a wider 
variety of qualitative and quantitative data. A 
feature of the qualitative data collection methods 
employed is the annual focus group interviews 
designed to gather feedback from representatives 
of client schools. As a result of the interviews, 
the RTLB cluster has received positive affirmation 
about its service and additionally, identified some 
key areas where its service could be strengthened. 
More importantly, it has developed a robust and 
practical system of review by blending information 
sources and methodology. Feedback collected 
can be fed directly into strategic planning and 
is proving useful in shaping practice. The focus 
group interviews are also benefiting participants 
by expanding their understanding of the RTLB 
role as well as enhancing professional networks. 
This article outlines the cluster's interpretation 
and implementation of authentic evidence-based 
practice (EBP). 
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THE CONTEXT OF THE REVIEWS 

The diversity of the Otaki Kapiti RTLB cluster 
and complexity of employment arrangements are 
important factors to consider when looking at the 
development of their review processes. The cluster 
is situated within a large semi-rural geographical 
area. It serves eighteen primary schools and three 
colleges between Paekakariki in the south and 
Manakau in the north. Within the schools there is 
considerable variance in terms of school culture 


and demographic profiles. The cluster employs ten 
RTLB in total. Due to job-share arrangements this 
equates to 7.5 full time RTLB positions. Included in 
the cluster is an RTLB Maori who primarily serves 
Maori immersion units and schools. The RTLB 
are hosted by five different schools and all RTLB 
are active members of the cluster management 
committee; this enhances ownership and 
responsibility for the annual reviews. While useful 
information on individual RTLB performance had 
previously been gathered, because of the diverse 
nature of the group, gaining an understanding 
of the effectiveness of the cluster as a whole had 
proven more difficult. 

THE DEVELOPMENT OF THE EFFECTIVENESS 
REVIEW MODEL 

Over the last eleven years, the RTLB had reported 
statistically to their management committee and 
the MOE. The majority of information provided 
related to referral numbers and types, year levels 
and the ethnicities of children on the RTLB roll. In 
2004, an Education Review Office (ERO) report 
on the RTLB service across the country found 
that a significant number of clusters could not 
provide evidence of effective practice. As one of 
forty clusters included in this review, the Otaki 
Kapiti RTLB cluster had received positive feedback 
from the ERO team. However the national finding 
prompted the RTLB cluster to reflect further on the 
way effectiveness reviews were conducted and to 
make modifications to its own systems. 

Initially, changes involved the RTLB collecting 
and presenting an expanded range of quantitative 
data to its management committee and the MOE. 
The most noticeable feature was recording and 
presenting the achievement (or non achievement) 
of casework goals within broad categories. In an 
attempt to improve the efficiency of collection 
processes, a range of different forms and templates 
was developed and trialled by the cluster. At this 
stage, individual casework and cluster effectiveness 
was largely measured by goals achieved within 
casework. 

In 2005, the cluster introduced qualitative 
reporting as an additional way of measuring 
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effectiveness. RTLB were asked to present a written 
narrative report to the management committee. 

The reports were based on a particular piece 
of case work and included information on the 
context, goals set, interventions and outcomes. 
They proved to be a useful way to demonstrate 
the complexity of RTLB case work as well as 
report on the collaborative consultative process 
that is central to the RTLB role. While the cluster 
found the combination of narrative and statistical 
reporting reassuring, as outcomes for students 
were often positive, the cluster was also aware 
that this type of reporting was not shaping practice 
or contributing to the overall improvement of 
the cluster as a whole. As a result, a decision 
was made in 2006 to evaluate the effectiveness 
of the RTLB service to local schools through a 
questionnaire. Specifically, the questionnaire 
would attempt to establish client expectation of the 
local RTLB service, identify parts of the service that 
were working well, and identify changes the RTLB 
cluster could make to improve practice. Obtaining 
information on client expectation was considered 
to be a vital prerequisite. A concern had developed 
that the clients understanding about the RTLB 
service may not be congruent with what the 
RTLB could offer; therefore interpreting feedback 
provided by teachers would remain unhelpful until 
further examination of client understanding was 
completed. 

The cluster established a working party to develop 
the questionnaire, however doubts arose about 
whether a questionnaire would provide the quality 
and type of information needed. This decision was 
based on previous unsuccessful questionnaires 
where the level and depth of response was low. 

The cluster hypothesised that teacher participants 
were not motivated to complete the survey 
because they may not have seen a direct benefit. 
Through discussion and further consultation, the 
concept of using focus groups as an alternative 
way of gathering the necessary data was promoted. 
This method of data collected aligns closely with 
the theory of evidence-based practice (EBP). 

Debate about EBP has long been occurring within 
academic circles. Interestingly, despite a lack of 
clarity about EBP, "almost everyone engaged in 
teaching and preparing educators would say that 
they employ evidence-based practice" (Siegrist, 
Leech, Bass & Patten, 2008, p. 147). At the heart of 
the argument was the question of what constitutes 
evidence and what does it actually mean to engage 
in evidence-based practice (Thomson & Anderson, 
in press). Walker (2004) suggests that the gold 
standard of evidence-based practice is that it is 
research-based, structured and clearly outlined 
in a prescribed manner. The RTLB cluster had 
begun to develop its own pragmatic definition of 


EBP, understanding the EBP to be the collection of 
robust and relevant evidence that proves directly 
useful in shaping practice to ensure improved 
outcomes for students, teachers and schools. 

THE FOCUS GROUP MODEL 

The RTLB cluster enlisted the support of an 
educational consultant to help plan and facilitate 
the focus group interviews and provide feedback. 
Focus group interviews are frequently used to 
promote and organise discussion with selected 
groups of individuals to gain information on their 
views and experiences of a topic (Gibbs, 1997). 
The cluster chose to use focus group interviews 
in this situation due to an understanding that 
relevant and rich data could be gained, with fewer 
constraints than questionnaires. The potential 
benefit to participants was also a contributing 
factor to the decision: it was hoped that feedback 
from focus group interviews would constitute 
genuine evidence-based practice rather than a 
mere exercise in consultation. 

The cluster identified key areas to be addressed 
within the focus group interviews. This included 
gaining the participants' understanding of the RTLB 
service in relation to: 

• Access to the service 

• Casework 

• Professionalism 

• Goal setting 

• The collaborative and consultative process. 

A series of questions was developed around these 
key areas and included: 

1. What is your understanding of the RTLB 
service? 

2. What were the main sources of that 
understanding? 

3. Which services provided by the RTLB have 
you found to be particularly useful/helpful? 

4. Have any of the services not been useful/ 
helpful? If so, what were they? 

5. Consider the cases that you believe had 
really good outcomes. What do you think 
were the main factors that contributed to 
their success? 

6. If you were to use three different adjectives 
to describe the RTLB, what would they be? 

7. Are there any general issues you would like 
to raise? 

Selection of participants has varied according to 
the numbers within the potential selection pool. 

In the first focus group interviews in 2007, the 
Special Education Needs Coordinators (SENCOs) 
from all cluster schools were invited. As this was a 
relatively small group, all could be accommodated 
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within two interviews. In 2008, classroom 
teachers were the selected group. As this was a 
large group, stratified sampling was used to select 
participants. Stratified sampling is based on a 
blend of both categorisation and randomisation as 
it divides a population into homogeneous groups 
based on similar characteristics (Cohen, Manion 
& Morrison, 2000). In this situation it allowed 
random selection within the following set criteria: 

• Classroom teachers who had worked with a 
RTLB during the previous year. 

• Classroom teachers from a range of year 
levels. 

• Classroom teachers from all cluster schools. 

A possible difficulty associated with focus group 
interviews is that individual participants can 
capture or dominate discussion, or inhibit others, 
so that a full range of perspectives is not gained 
(Arksey & Knight, 1999; Cohen, et al., 2000; 
Johnson & Christensen, 2000). The literature 
suggests that groups of six to twelve people is 
optimal (Arksey & Knight, 1999; Cohen, et al., 
2000; Gillham, 2000; Johnson & Christensen, 

2000) and in this review, groups of eight to ten 
were used. This was considered a number most 
useful in maximising the engagement of all 
participants. Groups were not too large to hinder 
those who do not like speaking in front of a 
group, and large enough so that a wide range of 
information and perspectives could be gathered. 

The third group interviewed in 2009 were senior 
managers from the secondary schools in the 
cluster. Again this was a small group so sampling 
was not required. 

OUTCOMES 

Focus group interviews have been completed 
for three years. There has been a good response 
from participants with most taking up invitations 
to attend interviews. In 2007, 80% of the invited 
SENCOs attended one of the two focus group 
interviews. Similar rates of participation were 
achieved in 2008 with classroom teachers. 

While establishing interview dates that suited 
all principals and members of college senior 
management was more difficult in 2009, 
interviews were well attended. Specific invitations 
to participants played a large part in the high 
participation rates achieved. 

Data collected from the interviews has given the 
RTLB cluster clear information of client expectation 
of the RTLB service. It became apparent that this 
was well matched to that of the RTLB cluster. 

The focus group interviews confirmed that the 
support provided to schools has been highly 


regarded. Furthermore, in three key areas, the 
participants provided a wide range of useful 
information regarding the factors that they believed 
contributed to the successful delivery of the RTLB 
service to local schools. 

Firstly, the clients identified a number of general 
factors that impact on success of the local service. 
Responses included were: 

• the ease of accessibility to the RTLB service; 

• the RTLBs' knowledge of school systems; 

• the practicality of interventions. 

Secondly, participants identified which RTLB 
services have been particularly useful across 
cluster schools. Typical responses were: 

• working to support teachers with group or 
whole class interventions; 

• training of teacher aides; 

• assistance with transitions of students from 
primary schools to secondary schools. 

Thirdly, participants were asked to consider RTLB 
cases where the outcomes had been particularly 
positive. Participants were then asked to identify 
what the main factors were that characteristically 
contributed to the successful outcomes of these 
cases. Typical responses included: 

• thorough data gathering and analysis; 

• liaison with families; 

• perseverance and consistency. 

The focus group interviews also provided specific 
feedback on areas that could be developed to 
enhance the RTLB service to the local schools. 

This included: 

• building on parent/caregiver understanding 
of the service; 

• acknowledging the difficulty of arranging 
meeting times; 

• providing professional development 
opportunities for client schools. 

The RTLB group has used the information gained 
from the focus group interviews to develop 
practice, for example, a parent/caregiver pamphlet 
has been produced. This pamphlet which has been 
published in both English and Maori, and provides 
information on how the RTLB service works and 
can be used by schools as they make decisions 
with parent/caregivers regarding an RTLB referral. 

Principals, SENCOs and teachers have increased 
their knowledge of the RTLB service as a result 
of being involved in the focus group interviews. 
They have also developed their understanding of 
the possibilities and potential of an RTLB referral 
through hearing of other educators' experiences. 
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The facilitator has prepared written evaluative 
reports which have been presented to the RTLB 
cluster management committee. Copies of the 
reports have been sent to all the cluster principals 
and to the MOE as an attachment to the annual 
report. 

ANALYSIS 

The cluster is confident in the accuracy, social 
validity and reliability of findings. Key factors 
involved in this were that the RTLB themselves 
were not included in the discussion groups. This 
provided anonymity and confidentiality for the 
participants. Having an independent facilitator 
with a sound knowledge of the RTLB model and 
the ability to shape the questions and responses 
through clarification and encouragement also 
promoted honest and genuine feedback. Stratified 
sampling ensured a transparent selection process 
which again contributed to the robustness of 
the process. The process ensured that individual 
RTLB were not professionally disadvantaged or 
embarrassed: findings were reported as themes 
relevant to the cluster as a whole. 

SUMMARY 

The cluster now has a comprehensive system of 
effectiveness review that combines quantitative 
and qualitative data collection. Statistical data 
continues to be collected. A software package 
and cluster training have enhanced the efficiency 
of the collection and presentation of the data. 
Qualitative data collection includes presentation of 
RTLB case studies to their management committee 
and the MOE. There remains the opportunity for 
informal feedback from the principals, SENCOs 
and teachers, however it is the advent of the 
focus group interviews that has rounded off data 
collection for the RTLB group. In previous years, 
the group had found that data collected was 
not easily transferred into a development plan. 
Feedback from the interviews (and other sources) 
can be directly linked into the RTLB cluster's 
annual strategic plan. The successful outcomes 
of the focus group interviews mean that the RTLB 
effectiveness reviews are purposeful, practical and 
of benefit to both RTLB and client schools. 

In conclusion, annual reviews allow the cluster 
to plan for improved service as well as report 
clearly to management committee, MOE and 
other interested groups such as ERO and fellow 
RTLB groups. Therefore the cluster's effectiveness 
reviews 'go beyond audit' and are an integral part 
of the cluster's effort to continually develop and 
improve their service to schools. 
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